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Executive Summary 


On November 2, 2015, San Francisco Department of Public Health's (SFDPH) 
Behavioral Health Services (BHS) Division launched the Assisted Outpatient 
Treatment (AOT) program (www.sfdph.org/aot). The program seeks to: 

• improve the quality of life of participants and support them on their path to 
recovery and wellness, 

• prevent decompensation, and 

• prevent cycling through acute services (e.g., psychiatric hospitalization) 
and incarceration. 

The program, authorized by San Francisco's Board of Supervisors in 2014, is one of 
a handful of County-led programs in California to support the primary intent and 
purpose of the Assisted Outpatient Treatment (AOT) Demonstration Project Act of 
2002 (Welfare and Institutions Code (WIC) §§5345-5349.5)—otherwise known as 
"Laura's Law"—to (a) identify persons with serious mental illness who are not 
engaged in treatment, (b) assess if there is substantial risk for deterioration and/or 
involuntary detention (under WIC §5150) which could be mitigated by provision of 
appropriate services, and (c) petition the court to order participation in such 
services if the individual is not able to be successfully engaged by other means. 

Prior to implementation it was anticipated that less than 100 individuals would be 
referred to AOT each year, with approximately 30 individuals meeting eligibility 
criteria and being served by program staff. A recent statewide assessment for AOT 
programs by the Treatment Advocacy Center suggests that San Francisco has a 
higher penetration rate for clients served through AOT than approximately 60% of 
other counties with available program data 1 . 

The San Francisco AOT Model is utilized as an intervention and engagement tool 
designed to assist and support individuals with serious mental illness. San 
Francisco's AOT program places an emphasis on promoting voluntary engagement 
by utilizing a strength-based and client-centered approach, as well as accessing an 
individual's natural support system (i.e., family and friends). The program has been 
constructed to employ principles of recovery and wellness, and has a particular 
focus on community-based services and multiple opportunities for an individual to 
engage in voluntary treatment before a court order is requested. This is the fourth 
report for the AOT program in San Francisco and covers implementation and 
outcome data across the entire three years of operation. 

The AOT Team 

Primary program services are provided by the AOT Care Team, in accordance with 
San Francisco Health Code §§4111-4119, which is currently comprised of a 
Program Manager, a masters level clinician, and two Team Members to provide 
peer and family support. The AOT Care Team conducts extensive outreach to: 

• assess appropriateness of a referral and eligibility for the program, 

• locate and engage Referred Individuals with local mental health resources, 

• encourage voluntary participation in treatment and engagement among 
Referred Individuals, and 


1 https://www.treatmentadvocacycenter.org/a-promising-start 
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• petition the court to order individuals into outpatient treatment when 
indicated. 

The Care Team works with Zuckerberg San Francisco General Hospital's Division of 
Citywide Case Management, Citywide AOT Team, to coordinate treatment for 
individuals that are court ordered into treatment and support individuals who have 
voluntarily agreed to services in linking to long term care. 

Calls to the AOT Care Team 

Between November, 2015 and December, 2018 the San Francisco AOT program: 

• Received 616 calls: 295 calls for referrals— predominantly from service 
providers and family members—and 321 requests for information. 

• Of the 295 incoming referrals during the evaluation period, 129 
unduplicated referred individuals were considered eligible for AOT 
participation and successfully contacted. The AOT team was unable to 
locate 8 individuals after multiple outreach attempts. 

Snapshot of All AOT Participants 

Individuals served by the AOT Care Team are at heightened risk of psychiatric 
hospitalization, incarceration, homelessness, and contact with Psychiatric 
Emergency Services (PES). In the 36 months (3 years) prior to being referred to 
the program: 

• 82% had at least one known inpatient psychiatric hospitalization in San 
Francisco. 

• 63% had at least one known incarceration in San Francisco. 

• 54% experienced at least one period of homelessness. 

• 81% had at least one known PES contact in San Francisco. 

Progress toward Outcomes for all AOT Participants 

Individuals in contact with AOT since the program's implementation showed 
overall reductions in PES contacts, psychiatric hospitalization, and incarceration. 
Specifically: 

• 74% of AOT participants were successful in reducing or avoiding PES 
contact. 

• 91% were successful in reducing or avoiding time spent in inpatient 

psychiatric hospitalization. 

• 88% were successful in reducing or avoiding time spent incarcerated. 

Well-Being, Social Functioning, and Independent Living Skills 

AOT Participants were surveyed, and respondents overwhelmingly reported a 
positive outlook on their future. 

• 77% of respondents feel confident that they can reach their treatment 
goals. 

• 86% of respondents feel hopeful about their future. 

Benefits of AOT engagement and case management were also reflected in the 
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survey responses of program participants 2 . 


• 64% of respondents believed that regularly meeting with a case manager 
will help them to find or maintain stable housing. 

• 67% of respondents believed that regularly meeting with a case manager 
will help them to maintain good physical health. 

• 67% of respondents believed that regularly meeting with a case manager 
will help them live the kind of life they want. 

Engagement and Treatment Promotion 

Between November, 2015 and December, 2018, 89 of the 129 individuals 
served by the AOT program voluntarily engaged in services. In total, 85 of 
the 129 individuals (66 percent) remained connected to a treatment 
provider at the time of this evaluation. In addition, the Citywide AOT Team has 
provided clinical case management services to 43 of these individuals who have 
voluntarily agreed to treatment or have been court ordered to participate in 
treatment (26 voluntary and 17 court ordered). 

AOT Care Team members and the Citywide AOT Team support clients in identifying 
their own treatment goals and plans for the future, and then building a treatment 
plan to support those goals. When asked about their interactions with AOT staff in 
the participant survey: 

• 82% of respondents reported that the AOT staff always treated them with 
respect. 

• 81% reported that the AOT staff always listened to their concerns about 
treatment. 

• 69% reported that AOT staff encouraged them to accept treatment 
voluntarily. 

Current Status and Disposition 

Over half (54%) of the individuals contacted by AOT whose cases are closed have 
achieved positive status outcomes—defined as being successfully connected to care 
and discharged—through their participation in the program. Additional dispositions 
include: 


• 17% of AOT participants continue to be active clients and have voluntarily 
engaged in treatment. 

• 2% are currently participating under a court order or settlement 
agreement. 

• 5% were unable to be located by AOT staff. 

• 15% were found to not meet the criteria for AOT participation upon 
assessment, but were offered support to access voluntary services. 

Snapshot of AOT Participants with Court Order 

Among the 17 court ordered AOT Participants during the evaluation period: 

• 14 individuals experienced at least one PES contact in the 36 months 
prior to AOT contact. 

• 15 individuals experienced at least one inpatient psychiatric 
hospitalization in the 36 months prior to AOT contact. 

• 10 individuals were incarcerated at least once in the 36 months prior 


2 Surveys were administered throughout the three years of program implementation. 
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to AOT contact. 

• 8 individuals had experienced homelessness in the 36 months prior to 

AOT contact. 

Progress toward Outcomes for AOT Participants with Court 
Orders 

The mixed methods evaluation detailed in this report is intended to fully address 
the regulatory program evaluation requirements of California Welfare and 
Institutions Code §5348(d). Findings in each of the fourteen mandated evaluation 
areas are summarized below, to the extent possible. 

• Over the course of the three-year evaluation period, a total of 17 
individuals were court ordered to participate in outpatient treatment and 
served by the AOT program. Of the 13 court ordered individuals who have 
been discharged, 9 were in independent or supportive housing situations at 
discharge, 3 were living with family, and 1 was in a residential program. Of 
the 4 active court ordered individuals, 1 is in independent housing, 2 are 
living with family, and 1 is in a stabilization bed. At the time of this report, 
only 1 of the 17 individuals have had their court orders end without being 
successfully connected to care. (§5348(d)(l)). 

• Since engaging in AOT through the court process, 15 of 17 individuals 
have successfully reduced or avoided time spent incarcerated 

(§5348(d)(2)). 

• One court ordered participant has stabilized enough to return to the 
workforce, successfully securing employment, and two others have 
secured volunteer stipended positions with Zuckerberg San Francisco 
General Hospital (§5348(d)(3)). 

• 14 of 17 individuals in the court process have successfully reduced or 
avoided time spent in inpatient psychiatric hospitalization since 
engagement in AOT (§5348(d)(4)). 

• The overall percentage of check-ins in which participants involved in the 
court process were in compliance with their treatment plans ranged from 
8 percent to 100 percent, with 15 out of 17 individuals maintaining 
compliance levels at or above 50 percent (§5348(d)(5)). 

• Since the program's implementation, the AOT Care Team has submitted a 
total of 11 court order renewals. At renewal, 9 out of the 11 participants 
volunteered to continue with the program (§5348(d)(6)). 

• Two individuals in the court ordered group were victims of violence 
during their time under court order (§5348(d)(7)). 

• 3 of 17 individuals who are court ordered perpetrated at least one act of 
violence over the course of the court process (§5348(d)(8)). 

• 6 of 17 individuals had instances of confirmed substance use over the 
course of the court process (§5348(d)(9)). 

• Throughout their participation in AOT, 100% of individuals who are court 
ordered had consistent contact with peer, clinical, and medical staff. Staff 
made at least one contact or attempted contact per week (§5348(d)(10)). 
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• At times during the evaluation period, AOT Participants were supported via 
increased frequency of court contact. There were no other enforcement 
mechanisms employed by staff to encourage compliance (§5348(d)(ll)). 

In the three years since the implementation of the AOT program, only one 
individual involved in the court process was asked to have increased 
contact with the court. 

• As a result of participation in AOT, individuals involved in the court process 
demonstrated an improved ability to function in their communities and 
build positive relationships with family members and others. Among the 17 
participants with a court order, 11 out of 17 show improved social 
functioning and 10 out of the 17 show improved independent living 
skills (§5348(d)(12)). 

• With support from AOT staff, individuals involved in the court process are 
able to successfully access community-based services vital to maintaining 
stable and independent living situations (§5348(d)(13)). During the 
evaluation period, 49 of the voluntary participants and 7 participants 
with a court order or settlement agreement were successfully linked to 
long-term care. 

• The majority of participants and family members reported positive 
perspectives on the approach to engagement by the AOT Care Team, and 
felt supported by program staff. 89% of individuals involved in the 
court process believed that regularly meeting with a case manager will 
help them live the kind of life they want, while the majority of the 14 
family members surveyed during this evaluation period reported feeling 
listed to and supported by AOT staff (§5348(d)(14)). 

Financial Analysis 

Overall, the implementation of AOT in San Francisco over the three-year evaluation 
period appears to have resulted in significant cost reductions related to 
psychiatric emergency services, inpatient psychiatric hospitalization, and 
incarceration, for the individuals who were served by the program. 

By examining average monthly utilization of these public systems for the 129 AOT 
participants included in this evaluation, and comparing those utilization rates for 
individuals in the time before and since engagement with AOT, implementation of 
the program appears to have resulted in an average estimated monthly savings 
of over $400,000. These savings are likely to have impacted both public city 
spending, as well as MediCal costs, depending on levels of individual coverage. 
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Introduction 


California State Assembly Bill 1421 (Thomson, Chapter 1017, Statutes of 2002) 
established the Assisted Outpatient Treatment Demonstration Project Act of 2002, 
more commonly referred to as "Laura's Law" (Welfare and Institutions Code (WIC) 
§§5345-5349.5). The purpose and intent of Assisted Outpatient Treatment (AOT) 
is to: 


• identify persons with serious mental illness who are not engaged in 
treatment, 

• assess if there is substantial risk for deterioration and/or involuntary 
detention (under WIC §5150) which could be mitigated by provision of 
appropriate services, and 

• petition the court to order participation in such services if the individual is 
not able to be successfully engaged by other means. 

Although established through a state measure, counties can decide whether and 
how—outside of select goals and service requirements—to implement the AOT 
program in their respective areas. 

The present report describes San Francisco's experience with AOT in the first three 
years of implementation, with a focus on the participants AOT engaged between 
November, 2015 and December, 2018. This is the fourth AOT Evaluation Report 
and provides information of use to local, state and regional government and law 
enforcement entities, as well as community, mental health, and other stakeholders. 


AOT Program Goals 

The Assisted Outpatient 
Treatment (AOT) model is 
designed to assist and support 
individuals with serious mental 
illness. The ultimate goals of the 
program are to improve the 
quality of life of participants 
and support them on their path 
to recovery and wellness, as well 
as prevent decompensation 
and cycling through acute 
services (e.g., psychiatric 
hospitalization) and 
incarceration. 


The Assisted Outpatient Treatment Program (AOT) 

In July 2014, San Francisco's Board of Supervisors authorized Assisted Outpatient 
Treatment as a response to the late Mayor Ed Lee's 2014 Care Task Force. 
Implemented November 2, 2015, and in operation now for over two years, the San 
Francisco Assisted Outpatient Treatment Model is utilized as an intervention and 
engagement tool designed to assist and support individuals with serious mental 
illness. The program has been constructed to employ principles of recovery and 
wellness, and has a particular focus on community-based services and multiple 
opportunities for an individual to engage in voluntary treatment. 

In San Francisco, the AOT program places an emphasis on promoting voluntary 
engagement by utilizing a strengths-based and client-centered approach, as well as 
accessing an individual's natural support system (i.e., family and friends). If after 
30+ days of engagement the staff is unable to successfully engage an individual in 
care, a petition to court order an individual into outpatient treatment may be 
pursued. This order uses the "black robe" effect—the symbolic weight of the 
court—to leverage an individual into care. 

Eligibility for AOT is initiated through a referral or petition request from a Qualified 
Requesting Party, as outlined in WIC §5346(b)(2). Qualified parties include an 
adult living with the individual, an individual's immediate family, treatment 
providers, or a parole or probation officer. Eligibility of Referred Individuals is then 
assessed by the clinical staff from the AOT Care Team. Individuals that appear to 
initially meet AOT criteria are subsequently engaged by the AOT Care Team and 
offered voluntary services. San Francisco's AOT model utilizes a multidisciplinary 
Care Team of clinical and peer-based services, as well as family support, to 
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Referred Individuals and their loved ones. The AOT Care Team conducts extensive 
outreach to: 

■ locate and engage Referred Individuals with local mental health resources, 

■ encourage voluntary participation in treatment and engagement among 
Referred Individuals, and 

■ petition the court to order individuals into outpatient treatment when 
indicated. 

Exhibit 1 provides an overview of this model below. 

Exhibit 1. Overview of Assisted Outpatient Treatment Program 


Criteria Not Met 


Resources and linkage to care 
assistance provided as indicated 


Not “Qualified Requesting 
Party" 


Referred Individual does 
not meet AOT criteria 


Petition Dismissed 


Intake and 
Investigation 



Petition Filed 



-| Court Hearing 



| Court-Ordered AOT 


Settlement Agreement 


Settlement Agreement 


Court Ordered 



Behavioral Health Services 


Coordinated care and case 
management 


The AOT Care Team is comprised of the Director (a psychologist with forensic 
experience) who oversees the general administration of the program, a licensed 
master's level Program Manager who oversees the day to day operations of the 
program, a master's level clinician, and two additional Team Members tasked with 
outreach, initial engagement and treatment, and client and family support. The 
AOT Care Team is housed within the San Francisco Department of Public Health's 
Behavioral Health Services. In addition to the AOT Care Team, a Clinical Team from 
Zuckerberg San Francisco General Hospital Division of Citywide Case Management 
("Citywide AOT Team") provides intensive clinical case management services, 
including psychiatric assessment and treatment coordination, for individuals court- 
ordered into treatment through AOT, and supports individuals who have voluntarily 
agreed to services in linking to long term care. The current Citywide AOT Team 
includes a Clinical Supervisor, one full-time Clinical Case Manager, a half-time Peer 
Specialist, and a part time Nurse and Psychiatrist. San Francisco relies on a highly 
collaborative model of service delivery and program improvement. Citywide AOT is 
in the process of hiring two additional full time clinical case managers. AOT staff 
work closely with service providers throughout the city, strengthening the referral 
network and modeling best practices. 

In order to remain accountable to the City of San Francisco and various stakeholder 
groups, AOT provides the San Francisco Health Commission and the AOT 
Implementation Committee with an annual update on the progress of the program, 
and a review of any related outcome data. Further, AOT contributes program data 
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to the Board of Supervisors via the MHSA 3-Year Integrated Plan and Annual 
Update, and provides any additional updates to the Board of Supervisors upon 
request and frequent updates to the Director of Behavioral Health Services. Outside 
of the city, the San Francisco AOT Team is building a network of family supports 
nationwide, has been instrumental in partnering with other counties that have 
adopted AOT, and has worked to initiate and sustain a quarterly conference call to 
share information and expertise. 

AOT Partners 

The following terms will be used to refer to the roles or relationships of various 
partners in San Francisco's AOT program: 

■ AOT Staff: The AOT Care Team and Citywide AOT Team defined below. 

■ AOT Care Team: The AOT Care Team within the Department of Public 
Health's Behavioral Health Services, focuses on program 
administration/reporting, intake, engagement and treatment of individuals, 
linkage to services, petitioning the court when indicated, and support 
through the legal process. The core AOT Care Team consists of the AOT 
Program Manager, clinician, and two Team Members to provide peer and 
family support. 

■ Citywide AOT Team: The Department of Public Health contracts with 
Zuckerberg San Francisco General Hospital Division of Citywide Case 
Management. This organization employs a team of mental health providers 
and a peer specialist to provide intensive clinical case management 
services, which include engaging individuals that are court ordered into 
outpatient treatment and supporting individuals who have voluntarily 
agreed to services in linking to long term care. This team is a Full Service 
Partnership funded through the Mental Health Services Act and provides a 
wide range of services to meet the unique needs of each individual (the 
services offered exceed the requirements outlined in WIC §5348). 

• AOT Participants/Contacted Individuals: All individuals contacted by 
the AOT Care Team, regardless of whether an appointment was missed. 

For the purposes of this report AOT Participants are classified as: 

■ Connected to Care and Discharged from AOT: Participants who 
have been successfully connected to care and are actively seeing a 
provider for treatment support. 

■ Accepted Treatment: Participants who have been connected to 
treatment and are in the process of ongoing AOT engagement. 

■ Ongoing Outreach: Participants who have yet to voluntarily accept 
treatment, for which the AOT Care Team continues outreach and 
engagement efforts. 

■ Petition Filed: Participants for whom the AOT Care Team has 
petitioned the court for engagement in outpatient treatment. 

■ AOT Families: Family, friends, and other close individuals who AOT 
Participants identify as providing them with support or assistance to AOT. 

Program Evaluation 

Behavioral Health Services (BHS) contracted with Harder+Company Community 
Research, in compliance with San Francisco Health Code §4118(c), to conduct an 
evaluation and in-depth analysis of individuals referred to AOT, the nature of 
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engagement in AOT activities, and the impact of AOT on mental health service 
utilization and other outcomes. The evaluation is intended to fully address 
regulatory program evaluation requirements and is summarized in an annual report 
each year of the program evaluation. 3 Now, having completed three full years of 
implementation, this final report seeks to provide insight into the program's long¬ 
term impact on all participants seen since the onset of the program. 

To fully understand the program's impact, Harder+Company and the AOT Care 
Team chose a multi-stakeholder, mixed methods evaluation which draws on the 
following data sources: 

■ Program Data: Data collected for all AOT Contacted Individuals since 
November 2015. This information includes AOT-related intake, initial, 
ongoing, and attempted contacts, background information, service linkage 
and use, and key events tracking. 

■ Participant Surveys: A paper-based questionnaire assessing thoughts 
about and experiences with AOT engagement and service linkage has been 
distributed to participants regularly throughout their time in AOT. 
Questionnaire completion was voluntary and no financial incentive was 
provided. AOT Participants submitted questionnaires to their case 
managers in a confidential and sealed envelope, who then passed them 
along to the evaluators at Harder+Company. 

■ Participant Interviews: A 20 minute interview conducted with AOT 
Participants with a court order. Interviews explored individuals' 
experiences with the AOT program, engagement, and service linkage, as 
well as thoughts on how the AOT program has impacted their lives. A total 
of three interviews were completed by Harder+Co and all were given a $25 
incentive for their time. 

■ Family Surveys: An eight item survey was distributed to members of the 
AOT Participants' families. The survey asked respondents about their 
satisfaction with AOT support, perceived benefits and challenges, as well 
as recommendations for the program. The Family Survey was administered 
online and via mail to consenting families in January 2018. No incentives 
were provided, and a total of 12 respondents completed the survey. 

■ Staff Interviews: A 30-minute interview conducted with members of the 
AOT Care Team and Citywide AOT Team to gather perspectives and 
reflections on program implementation, effectiveness, and larger social and 
organizational impacts. A total of 10 interviews were completed. 

■ Stakeholder Survey. A stakeholder survey was launched in November 
2017 to collect feedback on program implementation. The AOT Care Team 
helped identify partners who they felt could speak to the successes and 
challenges faced by the AOT program to-date, as well as any general 
recommendations they could share. A total of 22 stakeholders submitted a 
completed survey. 

■ Financial Analysis. In order to better understand the long-term financial 
implications of AOT in San Francisco, financial analysis was conducted to 
determine changes in the amount of public funding utilized for participants' 
incarceration, inpatient psychiatric hospitalization, and psychiatric 
emergency service use since the implementation of AOT. 


3 California Welfare and Institutions Code §5348(d) 
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Snapshot of AOT 
Participants 


The AOT program has established various methods by which Qualified Requesting 
Parties, as defined by WIC §5346 (b)(2) (e.g., an adult living with the individual, 
immediate family members, treatment providers, or a parole/probation officer), 
can make referrals and request information about the program. The program offers 
a toll-free number, local number, referral form and website, and general email 
address, and conducts outreach presentations to multiple stakeholders. Eligibility of 
referred individuals is then assessed by the clinicians on the AOT Care Team. This 
section of the report details information on all the individuals the AOT Care Team 
engaged throughout the entirety of the program - starting November 5, 2015 to 
December 31, 2018. 

Calls to the AOT Care Team 

Since November 2015 the AOT Care Team received 616 calls regarding the 
program. Of the 616 calls received, 321 were information only calls (52 percent) 
while the remaining 295 calls (48 percent) were referral requests, as detailed in the 
following exhibit. 

Exhibit 2. Type of AOT Calls (n=616) 



Information Only Referral 


Section Overview 

This section of the report 
details the following 
information: 

• Information on all 
calls received by the 
AOT Care Team 
during the evaluation 
period. 

• Demographic 
information on all 
active cases during 
the evaluation 
period. 

• Risk factors for all 
active cases during 
the evaluation 
period. 


The most common source of referral calls was providers (53 percent), while 39 
percent of the calls were from family members. Approximately 5 percent of referral 
calls were from persons who were not determined to be a Qualified Requesting 
Party (QRP). In these cases, the AOT Care Team worked with callers to identify 
potential QRPs to make the referral. 


12 





Exhibit 3. AOT Referring Parties (n = 295) 

53% 



Provider Family Non QRP Adult Parole 

Living 
with RI 


The majority of the calls made to the AOT program originated in San Francisco (76 
percent), while 16 percent of referrals originated from other counties in California. 
An additional 8 percent of referral calls originated from outside of California. 

Exhibit 4. Origin of Referral Calls by California County (n=295) 


San Francisco 


Out of State 


Contra Costa 

3% 

San Mateo 

3% 

Alameda 

™ 2% 

Marin 

M 1% 

Solano 

H 10/ 0 

Unknown 

tm io /o 

Los Angeles 

1% 

Santa Clara 

: 1% 

Sonoma 

1% 

Sacramento 

i 0% 

San Luis Obispo 

i 0% 

Santa Cruz 

i 0% 


Demographics of AOT Cases 

Of the 295 individuals referred to AOT since the onset of the program, 129 
unduplicated referred individuals were considered eligible for AOT 
participation and successfully contacted; AOT was unable to locate 8 individuals 
after multiple outreach attempts. 4 A small number of individuals were referred to 
the program multiple times 5 , in which case only their most recent engagement with 
AOT was included in the analysis. These AOT participants demonstrate considerable 
demographic diversity, as illustrated below. 

Since 2015 the AOT program served a majority of individuals who identified as 


4 In cases where an individual was determined not to be an eligible candidate, the referral source was 
provided with a consultation regarding alternative resources and, when indicated, family support was 
provided. 

5 8 individuals have two referrals to the AOT program, while one individual has 3. 
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male (59%) and 40 percent identified as female. Approximately 1 percent identified 
as non-binary and another 1 percent as transgender. 

Exhibit 5. Gender of Contacted Individuals (n=129) 


59% 

40% 

1 % 1 % 

Female Male Non binary Transgender - 

Male 



The greatest proportion (27 percent) of contacted individuals is between 26 and 35 
years of age. Overall, most program participants are under 46 years of age. 


Exhibit 6. Age Range of Contacted Individuals (n = 129) 


27% 


26% 



18-25 26-35 36-45 46-55 56-65 65+ 


The majority of individuals contacted identify as White (42 percent), followed by 23 
percent of individuals identifying as Black/African American, 16 percent as Asian, 
and 11 percent as Latinx. 
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Exhibit 7. Race of Contacted Individuals (n = 129) 


White 

Black/African American 
Asian 

Latinx 

Multi-Ethnic 

Native American 

Unknown 



1 % 

1 % 


Risk Factors 

Individuals in contact with AOT demonstrate significant needs, and are at 
heightened risk of psychiatric hospitalization, contact with Psychiatric Emergency 
Services (PES), incarceration, and homelessness. 

The exhibit below shows the percentage of program participants who experienced 
the above at least once during the 36 months prior to their participation in the 
AOT program. Most participants (82 percent) experienced at least one inpatient 
psychiatric hospitalization, and 81 percent of program participants experienced at 
least one PES contact in the 36 months prior to intake. Incarceration and 
homelessness were also common among the majority of participants. 

Exhibit 8. Psychiatric Hospitalization, PES Contact, Incarceration, and 

Homelessness in the 36 Months Prior to AOT Contact (n = 129) 


Homelessness 


Incarceration 


Psych Hospitalization 


PES 
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Progress Toward 
Outcomes for All AOT 
Participants 

To assess how the AOT program has impacted its participants, the following section 
compares participants' interactions with Psychiatric Emergency Services (PES), 
psychiatric inpatient hospitalization, and incarceration before and after their intake 
into the AOT program. The goal of this analysis is to track any changes in 
participants' contact with each of the three over time, before, during, and after 
each participant's engagement with the program and the AOT Care Team to better 
understand the long term impact of AOT as an intervention. 

In order to provide a holistic analysis of the program's impact, this section looks at 
the outcomes for all AOT participants since the beginning of the program's 
implementation, a total of 129 unduplicated individuals or 139 duplicated 
individuals. 6 Of these 129 individuals, 89 have voluntarily engaged in services. 


Psychiatric Emergency Services (PES) 

Among the participants analyzed, the number of individuals who experienced PES 
visits dropped from 105 individuals (81 percent) in the 36 months prior to their 
AOT participation, to 60 (47 percent) since contact with AOT. 

Exhibit 9. PES Occurrence 36 Months Prior and Since AOT Contact 
(n=129) 


81% 


36 Months Prior 


47% 



Since AOT 


Section Overview 

This section of the report 
details the following 
information: 

• Outcome information 
on the number of 
incidents and any 
changes in the 
average number of 
visits/days per 
month over time 

• Information on 
participants' well¬ 
being, social 
functioning, and 
independent living 
skills, as detailed by 
program participants 

• Information and 
quotes on the 
Citywide AOT Team's 
engagement with 
program participants 

• Information on all 
participants' current 
status and 
disposition 


In addition, average PES visits per month—for all participants—fell from 0.18 visits 
per month in the 36 months prior to AOT participation, to 0.07 after ending 
participation in AOT. This reduction in PES visits per month after exiting the 
program was also found to be statistically significant 7 , offering strong evidence 
that the observed change may be related to AOT participation. 


6 The unduplicated count is the number of unique individuals that participated in the AOT program. 
The duplicated count is the number of cases eligible to participate in the AOT program, including 
instances in which individuals participated multiple times in the program. 

7 Paired t(df)=128, p-value<0.05 





Exhibit 10. Average PES Visits per Month (n=129) 


0.20 

0.18 


0.07 



Before During After 


Psychiatric Inpatient Hospitalization 

The number of individuals experiencing at least one psychiatric inpatient 
hospitalization in the 36 months prior to intake dropped from 106 (82 percent) 
before AOT contact, to 40 (47 percent) since. 

Exhibit 11. Hospitalization 36 Months Prior, and Since AOT Contact 
(n=129) 


82% 


36 Months Prior 


47% 



Since AOT 


Although the total number of individuals experiencing an inpatient hospitalization 
decreased, average days per month hospitalized as a whole increased for the 
population of participants while they were actively engaged in AOT. However, this 
average increase from 0.96 days per month in the 36 months prior to AOT 
participation, to 2.27 days while actively engaged decreases to an average of 
0.29 days per month after participants have left the program. It is likely that 
these changes in the average number of days hospitalized over time is attributable 
to the participants' high need for services and long delayed access to much needed 
care, as well as the AOT Team's active assessment and advocacy with hospital staff 
and the court to ensure that participants receive the clinically appropriate level of 
care. Additionally, participants are encouraged to seek support and services when 
needed. Similar to the outcomes for PES, the decrease in the average days 
hospitalized per month after exiting the program was also found to be 
statistically significant 8 and further indicate the program's long-term 
effects on participants. 


Paired t (df)=126, p-value<0.05 
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Exhibit 12. Average Days Hospitalized per Month (n=129) 


2.27 


0.29 

Before During After 

Incarceration 

Lastly, when looking at participants' experiences with the justice system, the 
number of individuals with a recent incarceration dropped from 81 individuals (63 
percent) prior to AOT contact, to 51 (40 percent) since intake. 

Exhibit 13. Incarceration 36 Months Prior, and Since AOT Contact 
(n=129) 


0.96 


63% 


36 Months Prior 


40% 



Since AOT 


Average days incarcerated per month shows the biggest long-term decrease over 
time, from 2.31 days per month in the 36 months prior to intake to 1.34 days per 
month after exiting the program. While average days per month increased to 
3.13 while actively engaged in the program, this trend in average days per month 
is in line with the trend seen in days hospitalized per month and likely related to 
coordination of care and advocacy to ensure that legal dispositions take mental 
health needs into account. This decrease in average days per month after exiting 
the program was also found to be statistically significant 9 , again, offering strong 
evidence that these decreases can be attributed to participation in AOT. 


9 Paired t(df)=127, p-value<0.05 
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Exhibit 14. Average Days Incarcerated per Month (n=129) 


3.13 


2.31 



Before During After 


Summary of Outcome Attainment 

Exhibit 15, below, summarizes all of the above outcomes, including the observed 
changes over time in average PES visits per month, average days hospitalized per 
month, and average days incarcerated per month, before, during, and after 
engagement with the program and the AOT Care Team. 

Exhibit 15. Changes in Outcomes, 36 Months Prior, and Since AOT Contact 



36 Months 
Prior 

During 

Contact 

After Exiting 

Average PES Visits per Month 

0.18 

0.20 

0.07 

Average Days Hospitalized per Month 

0.96 

2.27 

0.29 

Average Days Incarcerated per Month 

2.31 

3.13 

1.34 


The scope of AOT's role in reducing negative outcomes is also evident when the 
averages referenced above are disaggregated. When each individual's monthly 
averages are compared pre and post-AOT contact, findings continue to suggest 
successful outcome attainment among the group. Individuals are considered 
to have attained a successful outcome by either reducing their monthly average of 
negative occurrences or—for those with no occurrences of a negative outcome prior 
to AOT contact—experiencing no occurrences of the negative outcome since AOT 
contact. 

Since the beginning of the program, 74 percent of AOT participants were 
successful in reducing or avoiding PES contact, 91 percent were successful 
in reducing or avoiding time spent in inpatient psychiatric hospitalization, 
and 88 percent were successful in reducing or avoiding time spent 
incarcerated. Additionally, at the end of December 2018, 68 unique program 
participants (53 percent) were closed and remained engaged in treatment. 
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Well-Being, Social Functioning, and Independent Living Skills 

The AOT program supports the basic needs and well-being of participants through 
provision of supportive resources, such as access to food, assistance applying for 
state identification cards, Supplemental Security Income (SSI), CalFresh (food 
stamps), and transportation assistance (e.g., Clipper Card). Peer staff provide 
additional support around activities of daily living, employment coaching, and other 
independent living skills as needed. Participants who were not able to function 
successfully in the community prior to AOT are often able to navigate complex 
processes with the additional support of the Care Team and Citywide AOT Team. 
When asked about their future via an AOT Participant Questionnaire, most 
participants had a positive outlook. Eighty-six percent of survey respondents 
reported that they feel hopeful about their future, and 77 percent reported feeling 
confident that they can reach their treatment goals. 

Exhibit 16. Outlook on Treatment Goals and Future (n=66) 



I am confident that I can reach my treatment goals n = 51 (77%) 

I feel hopeful about my future n = 57 (86%) 


For participants working with a case manager, the perceived impact of treatment 
on quality of life is measureable. Benefits of AOT engagement and case 
management were reflected in the survey responses of program participants. 64 
percent of respondents believed that regularly meeting with a case manager will 
help them to find or maintain stable housing, 67 percent believed that regularly 
meeting with a case manager will help them to maintain good physical health, and 
67 percent believed that regularly meeting with a case manager will help them live 
the kind of life they want. 

Exhibit 17. Perspectives on Meeting with a Case Manager (n=66) 


How much will regularly meeting with a case 
manager help you to... 

A Lot 

A Little/Not 
at All 

...find or maintain stable housing? 

n=42 (64%) 

n=13 (20%) 

...maintain good physical health? 

n=44 (67%) 

n=13 (20%) 

...live the kind of life you want? 

n=44 (67%) 

n = 13 (20%) 


Engagement and Treatment Promotion 

The Citywide AOT Team provides intensive clinical case management for court- 
ordered AOT Participants, and provides support for individuals who agree to 
voluntary services, linking them to long term care. This support is critical to clients' 
long term success. As one Care Team member explains, "[the Citywide AOT Team] 
really meets each person they work with wherever they are at to engage and 
stabilize them... Once stable in the community, the team then coordinates a smooth 
and successful transition to a long term provider." 

According to the Citywide AOT Team, achieving benefits for program participants 
requires being persistent and client centered. One staff member shared how the 
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unique nature of AOT allows them to successfully engage people who would 
otherwise remain disconnected: 

"For a lot of our clients the systems have already tried, they've been trying 
for years to engage these individuals and reduce hospitalization, reduce 
incarceration, and reduce crisis contacts. But I think the uniqueness of this 
program, having additional peer support, having low caseloads, the 
flexibility to do the amount of outreach that we're doing, I really think this 
program is changing and improving the lives of these individuals." 

Since the beginning of AOT implementation, the Citywide AOT Team has provided 
case management support to 43 individuals who have voluntarily agreed to 
treatment, 17 of whom have been court ordered court ordered. Of these, 9 
voluntary and 4 court ordered individuals are still receiving services (as of 
December 2018) from the Citywide Team that are designed to stabilize them in 
preparation for linkage to long term services. Fourteen of the voluntary participants 
and 5 court ordered participants were successfully linked to long-term care. Only 1 
court ordered individual was not successfully linked to long-term care due to 
refusing ongoing services and referrals. 

Besides persistence, respect for participants was mentioned by staff as central to 
the success of the program. AOT Care Team members emphasized that client 
engagement begins with supporting the client in identifying their own treatment 
goals and plans for the future, and then building a treatment plan to support those 
goals. This emphasis on respect is validated by participants' responses in the 
participant survey. Eighty-two percent of respondents reported that staff have 
always treated them with respect, and 81 percent reported that the staff always 
listened to their concerns about treatment. Sixty-nine percent also noted that they 
were always encouraged to accept treatment voluntarily, which may be related to 
the open conversations that the AOT Team has with participants regarding the 
possibility of a court order. 

Exhibit 18. Interactions with the AOT Team 


The AOT Team always... 


...treated me with respect 

n = 23 (82%) 

...listened to my concerns about treatment 

n = 21 (81%) 


...encouraged me to accept treatment voluntarily n = 18 (69%) 


Current Status and Disposition 

As of the end of 2018, when looking at non-active cases, 54 percent of individuals 
in contact with AOT achieved positive status outcomes—defined as being 
successfully connected to care and discharged—through their participation in the 
program. This total includes 12 individuals who were court ordered and successfully 
connected to care. Twenty-five individuals (19 percent) were active AOT cases at 
the time of data collection. In cases where an individual does not meet the legal 
eligibility for AOT, staff support the referral entity in identifying other appropriate 
resources. In total, 15 percent of individuals were found to not meet the criteria of 
participation for the program after initial contact was made. A full accounting of 
dispositions for the 129 individuals included in the evaluation is presented in 
Exhibit 19 below. 
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Exhibit 19. Disposition of Contacted Individuals (as of 12/31/18) 


Disposition 

Closed - court order not granted 

n = 2 (2%) 

Closed - court order ended / withdrawn 

n = 12 (9%) 

Closed - other 

n=8 (6%) 

Closed - successfully connected / re-engaged in services 

n=49 (38%) 

Closed - criteria not met - client outreached 

n = 19 (15%) 

Closed - referral withdrawn 

n = 7 (5%) 

Closed - unable to locate 

n = 7 (5%) 

Open - active case 

n = 21 (16%) 

Open - court ordered / settlement agreement 

n=4 (3%) 


Conservatorship 

Sixteen individuals (12 percent) that AOT outreached and engaged with since the 
beginning of the program were determined to need a higher level of care than the 
AOT program provides. With the support of AOT staff, the Department of Aging and 
Adult Services, and the courts, these 16 individuals were placed on an LPS 
Conservatorship, providing them the safety and stability needed to ensure their 
recovery. Conservatorship services are provided only for individuals who meet the 
definition of grave disability, which includes being unable to provide for their basic 
needs (i.e. food, clothing, and shelter) due to a mental disorder. These services 
may include individualized treatment, supervision, and an involuntary medication 
order. Of the 16 individuals conserved, 13 remain housed in the community, while 
3 now reside in locked facilities which provide them with a higher level of care and 
personal safety appropriate for their needs. 
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Snapshot of Court 
Ordered AOT Participants 


In cases where the AOT Care Team is unable to successfully engage individuals in 
voluntary treatment, petitions may be filed for court ordered treatment. These 
individuals who are court ordered are subject to enhanced monitoring by the AOT 
Care Team, and present significant needs and challenges. 

From the beginning of the program to present, the AOT Care Team has filed a total 
of 31 petitions for 20 unique individuals, 17 of which moved forward either as a 
settlement agreement 10 or a court order to participate in the AOT program. * 11 This 
snapshot and all the exhibits subsequently shown in this section of the report focus 
on the 17 individuals who moved forward with a settlement agreement or were 
court ordered to participate in the program, henceforward referred to as individuals 
who were court ordered into the program. 

Exhibit 20. Contacted Individuals and Court Orders (n=129) 


87% 



Not Court Ordered Court Ordered 


Section Overview 

This section of the report 
details the following 
information: 

• Demographic 
information on all 
court ordered 
individuals during the 
evaluation period 

• Risk factors for all 
court ordered 
individuals during the 
evaluation period 


Additionally, since the beginning of implementation, the AOT Care Team has 
submitted a total of 11 court order renewals. At renewal, 9 out of the 11 
participants volunteered to continue with the program, and one renewal was 
withdrawn. Only one individual had to be court ordered into treatment upon 
renewal. 


10 A Settlement Agreement is when an individual accepts services once a petition is filed with the 
court. 

11 Of the petitions filed that did not move forward, one case withdrew and later voluntarily engaged 
in services, and the other had their court order dismissed. 
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Demographics 

Among court ordered AOT Participants, 8 (47 percent) identify as female, and 9 (53 
percent) identify as male. 

Exhibit 21. Gender of Individuals who are court ordered (n=17) 

53% 

47% 

Female Male 




Since the beginning of AOT implementation, the majority (71 percent) of court 
ordered participants have been younger than 45 years of age. Three participants 
are between 46 and 55 years of age, one is between 56 and 65 years of age, and 
one is older than 65 years of age. 

Exhibit 22. Age of Individuals who are court ordered (n = 17) 


35% 



The highest percentage of court ordered individuals identified as White (35 
percent), while four court ordered participants identify as Black/African American 
(24 percent) and two as Latinx (12 percent). 
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Exhibit 23. Race/Ethnicity of Individuals who are court ordered (n = 17) 


Multiracial 

Native American 

Asian 

Latinx 

Black/African American 

White 



Risk Factors 

Fourteen of the 17 individuals who were court ordered during the evaluation period 
experienced at least one PES contact in the 36 months prior to AOT contact, and 15 
experienced at least one inpatient psychiatric hospitalization. Ten individuals who 
are court ordered were incarcerated at least once in the 36 months prior to AOT 
contact, and 8 had experienced homelessness. 

Exhibit 24. PES Contact, Psychiatric Hospitalization, Incarceration, and 

Homelessness for Individuals who are court ordered in the 36 
Months Prior to AOT Contact (n=17) 


PES 


Psych Hospitalization 


Incarceration 


Homelessness 
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Of the 17 court ordered cases included during the evaluation period, the majority 
were either living independently, living in supportive housing, or living with family 
(88 percent). Only one individual is in a stabilization bed. 

Exhibit 25. Housing for Individuals who are Court Ordered (Current or at 
Time of Discharge) 


Housing Status 

Total 

Independent 

6 

Supportive Housing 

4 

With Family 

5 

Stabilization Bed/Room 

1 

Residential Program 

1 
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Progress Toward 
Outcomes for Court 
Ordered AOT Participants 


As with outcomes for all AOT Participants in the previous section, outcome data for 
all 17 individuals who are court ordered or reached a settlement agreement were 
examined across the three years since the program's implementation and are 
included as a part of the following analysis. Due to the small number of individuals 
who are court ordered, the subsequent analysis cannot show statistical 
significance. However, notable reductions in negative outcomes are highlighted 
where relevant. 

Psychiatric Emergency Services, Inpatient Psychiatric 
Hospitalization, and Incarceration 

The number of individuals who experienced PES contact dropped from 14 in the 36 
months prior to AOT participation, to 10 since being court ordered. 

Exhibit 26. PES Occurrence 36 Months Prior, and Since AOT Court Order 
(n=17) 


82% 


36 Months Prior 


59% 



Since Court Order 


The number of individuals with a recent inpatient psychiatric hospitalization 
dropped moderately from 15 in the 36 months prior to AOT participation, to 14 
since being court ordered. 

Exhibit 27. Hospitalization 36 Months Prior, and Since AOT Court Order 
(n=17) 


88 % 


36 Months Prior 


82% 



Since Court Order 
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Finally, the number of individuals with a recent incident of incarceration changed 
from 6 in the 36 months prior to AOT participation to 3 since being court ordered. 

Exhibit 28. Incarceration 36 Months Prior, and Since AOT Court Order 
(n=17) 


59% 


36 Months Prior 


29% 



Since Court Order 


Due to the small size of the court ordered AOT population, exploring changes in the 
averages of PES visits, days hospitalized, and days incarcerated is not particularly 
meaningful at the aggregate level. Instead, it is helpful to examine changes in 
these occurrences over time at the individual level. Exhibit 30, below, highlights 
each of the court ordered participants' changes in average occurrences per month, 
from 36 months prior to AOT contact, to post court order. 

Exhibit 29. Changes in Average PES Visits per Month, Days Hospitalized 
per Month, and Days Incarcerated per Month, by Participant 



PES Visits per Month 

„ „ . Positive 

Pre Post _ . 

Outcome 

Days Hospitalized per 
Month 

„ _ Positive 

Pre Post _ . 

Outcome 

Days Incarcerated per 
Month 

_ _ . Positive 

Pre Post _ . 

Outcome 

Participant 1 

0.08 

0.00 

✓ 

0.39 

0.00 

✓ 

0.00 

0.00 

✓ 

Participant 2 

0.19 

0.05 

✓ 

5.50 

0.60 

✓ 

0.03 

2.15 


Participant 3 

0.19 

0.00 

✓ 

1.36 

0.00 

✓ 

0.00 

0.00 

✓ 

Participant 4 

0.94 

0.00 

✓ 

0.11 

0.00 

✓ 

0.17 

0.00 

✓ 

Participant 5 

0.08 

0.07 

✓ 

0.44 

0.86 


5.03 

0.00 

✓ 

Participant 6 

0.00 

0.00 

✓ 

0.00 

0.00 

✓ 

0.00 

0.00 

✓ 

Participant 7 

0.33 

0.23 

✓ 

1.00 

8.00 


6.44 

0.00 

✓ 

Participant 8 

0.08 

0.00 

✓ 

1.50 

0.00 

✓ 

0.00 

0.00 

✓ 

Participant 9 

0.11 

0.00 

✓ 

1.03 

1.40 


0.11 

13.47 


Participant 10 

0.22 

0.00 

✓ 

0.67 

0.00 

✓ 

5.44 

0.00 

✓ 

Participant 11 

0.08 

0.00 

✓ 

0.83 

0.00 

✓ 

0.50 

0.00 

✓ 

Participant 12 

0.00 

0.00 

✓ 

0.00 

0.00 

✓ 

17.86 

0.00 

✓ 

Participant 13 

0.11 

0.00 

✓ 

0.89 

0.00 

✓ 

0.00 

0.00 

✓ 

Participant 14 

0.00 

0.00 

✓ 

1.22 

0.00 

✓ 

0.00 

0.00 

✓ 

Participant 15 

0.06 

0.00 

✓ 

1.67 

0.00 

•/ 

0.11 

0.00 

✓ 

Participant 16 

0.03 

0.00 

✓ 

3.28 

0.00 

✓ 

0.00 

0.00 

✓ 

Participant 17 

0.08 

0.00 

✓ 

2.89 

0.00 

•/ 

0.56 

0.00 

✓ 


As defined in a previous section of this report, individuals are considered to have 
attained a successful outcome by either reducing their monthly average of negative 
occurrences or—for those with no occurrences of a negative outcome prior to AOT 
contact—experiencing no occurrences of the negative outcome since AOT court 
order. During the evaluation period, all 17 ordered participants were successful in 
reducing or avoiding PES contact, 14 were successful in reducing or avoiding time 
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spent in inpatient psychiatric hospitalization, and 15 were successful in reducing or 
avoiding time spent incarcerated. 

Treatment Plan Progress and Adherence 

The following exhibits detail various outcomes for individuals who are court ordered 
as tracked by the AOT Care Team since the program's implementation. 

Exhibit 30. Court Ordered Treatment Plan Adherence 



% of 

Check-Ins 

Compliant 

with 

Treatment 

Confirmed 

Substance 

Use 

Violence 

Victim 

Violence 

Perpetrator 

Participant 1 

98% 


- 

- 

Participant 2 

32% 

✓ 

- 

✓ 

Participant 3 

62% 

•/ 


■/ 

Participant 4 

92% 

- 

- 

- 

Participant 5 

89% 

■/ 

» 

- 

Participant 6 

58% 

- 

- 

- 

Participant 7 

62% 

- 

•/ 

■/ 

Participant 8 

64% 

✓ 

- 

- 

Participant 9 

8% 

* 


- 

Participant 10 

100% 

- 

- 

- 

Participant 11 

100% 

* 

«>■ 

- 

Participant 12 

100% 

✓ 

- 

- 

Participant 13 

100% 

✓ 


- 

Participant 14 

50% 

- 

- 

- 

Participant 15 

88% 

* 

- 

- 

Participant 16 

75% 

- 

- 

- 

Participant 17 

100% 


s 

- 
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Exhibit 31. Court Ordered Enforcement Mechanisms 



Order to 
appear 

Increased 
contact 
with the 
court 

Judge 

feedback 

Court 

order 

extended 

Participant 1 

✓ 

- 

✓ 

✓ 

Participant 2 

✓ 

✓ 

✓ 

✓ 

Participant 3 

✓ 

- 

✓ 

- 

Participant 4 

- 

- 

- 

- 

Participant 5 

- 

- 

✓ 

- 

Participant 6 

✓ 

- 

✓ 

- 

Participant 7 

- 

- 

- 

✓ 

Participant 8 

✓ 

- 

✓ 

✓ 

Participant 9 

✓ 

- 

- 

- 

Participant 10 

- 

- 

✓ 

- 

Participant 11 

- 

- 

✓ 

- 

Participant 12 

- 

* 

✓ 

- 

Participant 13 

- 

- 

✓ 

s 

Participant 14 

- 

- 

✓ 

s 

Participant 15 

- 

- 

- 

- 

Participant 16 

✓ 

-■ 

- 


Participant 17 

- 

- 

✓ 

■ 


Exhibit 32. Court Ordered Social and Living Skills 



Improved 

Social 

Functioning 

Improved 
Independent 
Living Skills 

Participant 1 

- 

- 

Participant 2 

- 

V 

Participant 3 

S 

S 

Participant 4 

S 

V 

Participant 5 

s 

s 

Participant 6 

- 

V 

Participant 7 


- 

Participant 8 

- 

- 

Participant 9 

- 

- 

Participant 10 


V 

Participant 11 

s 

s 

Participant 12 

V 

V 

Participant 13 

V 

V 

Participant 14 

V 

V 

Participant 15 

V 

- 

Participant 16 

- 

- 

Participant 17 


- 
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Exhibit 33. Court Ordered Participant Survey Results 



Participant is confident 
they can reach their 
treatment goals 

Participant is hopeful 
about their future 

Participant received help 
to navigate the AOT 
program and courts when 
they needed it 

Participant 1 

V 

V 

V 

Participant 2 

V 

S 

V 

Participant 3 

V 

S 

s 

Participant 4 

V 

•/ 

•/ 

Participant 5 

- 


s 

Participant 6 

V 

•/ 

V 

Participant 7 

V 


V 

Participant 8 

V 

•/ 

V 

Participant 9 

N/A 12 

N/A 

N/A 

Participant 10 

V 

- 

- 

Participant 11 

N/A 

N/A 

N/A 

Participant 12 

N/A 

N/A 

N/A 

Participant 13 

y' 

S 

- 

Participant 14 

N/A 

N/A 

N/A 

Participant 15 

N/A 

N/A 

N/A 

Participant 16 

N/A 

N/A 

N/A 

Participant 17 

N/A 

N/A 

N/A 


Well-Being, Social Functioning, and Independent Living Skills 

Before participants enter AOT, many are in need of extensive support to stabilize. 
However, interventions for hard-to-reach individuals often require a prolonged 
outreach process, as was described by one family member who participated in the 
family survey: 

"AOT and I kept in contact and repeatedly tried to get [my child engaged 
in services], [AOT Care Team staff] set aside a time every week hoping he 
would show up. He finally went in and they began working with him and 
got him back on his medicine. It took us 2 years but without the AOT team 
he could have been killed." 

Interventions, like the one described above, are only possible because of the way 
AOT is structured, with an emphasis on client-centered outreach, flexibility, and 
meeting individuals where they are. As a result of this consistent outreach, the AOT 
Team is able to successfully engage and stabilize individuals in a way that 
positively impacts their well-being. 

As mentioned previously, AOT staff also support participants as they navigate 
different systems of care and establish a foundation for a more normalized life. One 
court ordered participant explained how helpful AOT Care Staff were in helping 
them accomplish several things, specifically, "[AOT Care Team staff] made me feel 
safe. They were worried about me being on the streets. [Since being court ordered] 
I haven't been on the street as much, I haven't been homeless... [They] helped me 
get my ID, and [they're] working on helping me get my SSI." This individual's 


12 N/A is indicated for individuals who have not completed a participant questionnaire 
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experience illustrates the gradual but impactful ways that stabilization can occur 
and the client-centered nature of outreach and engagement. In the time since this 
interview, this individual has remained housed and actively engaged in services. 
Both their social functioning and independent living skills have shown a marked 
increase since engagement began, and continue to improve. 

This sort of stabilization is most easily achieved when participants willingly engage 
in services, but not all do. Nonetheless, AOT's practice of consistent and persistent 
outreach and engagement allows them to reach otherwise disconnected individuals. 

Benefits of AOT's unique engagement structure for individuals who are court 
ordered were also reflected in participants' survey responses. Nearly all (88 
percent) of respondents who were court ordered believed that regularly meeting 
with a case manager will help them to find or maintain stable housing. 89 percent 
believed that regularly meeting with a case manager will help them to maintain 
good physical health, and 89 percent believed that regularly meeting with a case 
manager will help them live the kind of life they want. 


"AOT's diligence and care 
has literally been life 
saving for my sister." 

-Family Member 


Exhibit 34. Court Ordered Perspectives on Meeting with a Case Manager 


How much will regularly meeting with a case 
manager help you to... 

A Lot 

A Little/Not 
at All 

...find or maintain stable housing? 

88% 

12% 

...maintain good physical health? 

89% 

11% 

...live the kind of life you want? 

89% 

11% 


Consistent AOT engagement with individuals who are court ordered appears to also 
positively impact participants' outlooks on their future. When surveyed, nearly all 
(90 percent) respondents who were court ordered reported feeling confident that 
they can reach their treatment goals, and 90 percent reported that they feel 
hopeful about their future. 

When interviewed, one court-ordered individual discussed their desire to be able to 
work again in the future, and another discussed concrete employment goals that 
they are working toward. To date, one court ordered participant has stabilized 
enough to return to the workforce, successfully securing employment. 
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Financial Analysis 


Overview and Method 

As previously mentioned, the overarching goals of AOT implementation include the 
reduction of particular client outcomes, including use of psychiatric emergency 
services, inpatient psychiatric hospitalization, and incarceration (§5348(d)). In 
addition to their impacts on AOT participants and their families, these outcomes 
also have financial implications for the public systems that support them. At the 
time of this report, a single instance of PES contact is associated with a cost of 
$2,962, one day of inpatient psychiatric hospitalization carries a cost of $2,837, 
and a day spent in San Francisco jail comes at a cost of $245. In the case of PES 
contact and inpatient psychiatric hospitalization, these costs are at times absorbed 
by MediCal coverage, if the individual is a beneficiary. Whether the impact is on 
MediCal or city spending, even slight reductions in AOT participants' contact with 
these systems can have significant financial implications. 

In order to better understand these financial implications, and in compliance with 
San Francisco health code (§4118(c)), this three-year evaluation report includes an 
analysis of the total estimated costs associated with the three outcomes noted 
above, for all 129 AOT participants. Data on each of these individuals' contact with 
PES, psychiatric hospitalization, and the jails over time was used to estimate the 
total average monthly cost of service for each. For the purposes of analysis, each 
participant's average monthly utilization of these services was calculated both for 
the time before their engagement with AOT, and the time since their engagement. 
These monthly averages were then summed across the entire AOT participant pool, 
creating an estimate of overall monthly costs before clients engage with AOT, and 
after. This calculation serves as a reliable estimate of the actual cost implications 
for the public systems supporting AOT participants. 

Financial Implications 

Overall, the implementation of AOT in San Francisco over the three-year evaluation 
period appears to have resulted in significant cost reductions related to 
psychiatric emergency services, inpatient psychiatric hospitalization, and 
incarceration, for the individuals who were served by the program. 

Prior to their engagement with AOT, these 129 individuals combined carried a total 
average cost of $67,139 per month for psychiatric emergency services (22.7 
instances per month total or 0.18 instances per person), $345,799 per month for 
inpatient psychiatric hospitalization (121.9 days per month total or 0.94 days per 
person), and $72,422 per month for time spent incarcerated (295.4 days per 
month total or 2.29 days per person). In total, utilization of these three 
systems averaged over $485,000 per month for the pool of 129 
participants, before their engagement in AOT. 

When examining average costs in the time since these participants engaged in AOT 
(including both their time during the program and after case closure), these 129 
individuals carried a total average cost of $50,286 per month for psychiatric 
emergency services, $29,452 per month for inpatient psychiatric hospitalization, 
and $2,007 per month for time spent incarcerated. In total, utilization of these 
three systems averaged $81,745 per month for the pool of AOT participants, when 
looking at the time since they became engaged with AOT. This represents an 
average estimated monthly savings of $403,614, (approximately 83 
percent). 


Exhibit 35. Average estimated monthly cost of psychiatric emergency 
services before and since engagement with AOT 


$67,139 



Monthly Cost Pre-AOT 


$50,286 


Monthly Cost Since AOT 


Exhibit 36. Average estimated monthly cost of inpatient psychiatric 
hospitalization before and since engagement with AOT 


$345,799 



Monthly Cost Pre-AOT 


$29,452 


Monthly Cost Since AOT 


Exhibit 37. Average estimated monthly cost of incarceration before and 
since engagement with AOT 


$72,422 



Monthly Cost Pre-AOT Monthly Cost Since AOT 
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Exhibit 38. Average total estimated monthly cost of psychiatric 

emergency services, inpatient psychiatric hospitalization, and 
incarceration before and since engagement with AOT 


$485,360 



35 








Benefits for Family 
Members 


With the understanding that participants benefit from strong family support during 
their journey to recovery and wellness, San Francisco prioritized family 
engagement when planning and implementing AOT. Consequently, as part of the 
AOT process, families are engaged and offered education and support, including 
information about eligibility, benefits, limitations, and opportunities of the program. 
The fourteen family members surveyed during this evaluation period learned about 
AOT and were referred to the program through multiple avenues, including a 
therapist/case manager/mental health agency, an AOT-referred family member, an 
attorney/judge, the National Alliance on Mental Illness (NAMI), homeless services, 
and even an article about AOT. 

As a result of the support and education AOT has provided, many families reported 
that they had more or better access to resources and a better understanding of the 
symptoms their loved ones experience. Half reported a better understanding of 
what AOT is and how it can help their family member, and a better understanding 
of the mental health system as a whole. Some reported appreciating the way the 
AOT Team listened and talked to them. For example, one family member described 
how AOT has supported them and helped them to support their loved one, saying 
"Dealing with my [loved one's] situation is hands down, the most difficult challenge 
/obstacle in my life... For me personally, it has been a tremendous help to be able 
to talk to someone [from the AOT staff], I am eternally grateful." 

AOT staff repeatedly emphasized two ways that AOT is uniquely situated to support 
families—the program's ability to empower family members of at-risk individuals 
and the support it provides directly to family members. One of the biggest benefits 
for families, as reported by both family members and AOT staff, is the ability for 
families to directly refer their loved ones to the program. As one AOT staff member 
explained, it empowers families "to feel like they have agency, they have a voice, 
and they have an avenue to get their loved ones support when nothing else they've 
tried has really worked." 

Additionally, reflecting on their experiences working with clients and their families, 
AOT staff acknowledged the stress that families face when supporting the long 
term recovery and success of program participants, highlighting the second major 
benefit for families. As one AOT staff member explained: 

"[AOT] supports family members, helping them understand their loved 
ones' diagnosis and that this is an ongoing journey. We teach them new 
coping skill tools to use... we find out what their needs are and what their 
concerns are and we try to accommodate them with supportive help, with 
agencies, and with referrals out to other family support groups ." 

This support, staff said, is critical, because investing time to educate family 
members on mental health, what it means to have a mental illness, and how to 
strengthen participants' support systems can result in improved outcomes for 
program participants. 


"AOT saved my family in a 
way I can't explain. I'm 
very grateful to this 
amazing program and 
team." 

-Family Member 
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Strengthening Partnerships 


By strengthening the network of providers that AOT works with, the AOT Care 
Team is able to better identify and connect participants with appropriate treatment 
options. In November 2017, the AOT Care Team identified a number of providers 
and other stakeholders they felt could speak to the successes and challenges faced 
by the AOT program to date. These stakeholders were asked to complete a survey 
where they shared program feedback, their observations about the impact of AOT 
on Referred Individuals, and general feedback about AOT's role as part of the 
larger mental health system in San Francisco. 

When asked, most stakeholders reported observing numerous positive changes in 
Referred Individuals. Eighty-six percent of respondents reported at least some 
increased interaction with mental health services, 73 percent reported at least 
some increase in more positive interactions with mental health services, and 68 
percent reported at least some increase in mental health stability. 68 percent also 
reported at least some increase in housing stability, while 64 percent of 
stakeholders surveyed observed at least some improvement in Referred 
Individuals' functioning. 

Exhibit 39. Positive Changes Observed in Referred Individuals 


To what extent have you observed the following positive 
changes in Referred Individuals since beginning AOT? 

At least some 

Increased interaction with mental health services 

n = 19 (86%) 

More positive interactions with mental health services 

n = 16 (73%) 

More stable mental health 

n = 15 (68%) 

More stable housing 

n = 15 (68%) 

Improved functioning 

n = 14 (64%) 


Through strong collaboration with providers across San Francisco, AOT offers 
access to care and support tailored to each participant's unique needs, while 
simultaneously strengthening partnerships throughout San Francisco's system of 
care. One stakeholder shared that collaboration begins during outreach, saying 
"AOT Case Managers have increased relationships with multiple partners, including 
street medicine and crisis services, and get folks medicated and evaluated quickly." 
Collaboration extends beyond San Francisco's network of providers, as well, with 
one stakeholder explaining "I think the collaboration between DPFI, the City 
Attorney's Office, the clinicians, the Public Defender's Office and the courts is very 
successful. I think we all feel we are working to together for the best interest of the 
individual." 


"The AOT program has 
brought agencies together 
and made for better 
working relationships that 
were not there prior." 


Both AOT staff and stakeholders discussed a desire to expand collaboration to -Stakeholder 

better support participants on their journey to recovery and wellness, something 
AOT is actively pursuing through increased community outreach and education. 

One AOT staff member explained, "we're continuing to do presentations in the 
community, making sure to really get that information out there to people about 
what AOT is, what we can do, how we're here to support both referred individuals 
as well as community providers, and that we really want to work together." 
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Conclusion 


Summary of Findings 

During the three-year evaluation period, AOT Participants demonstrated overall 
reductions in PES visits, inpatient psychiatric hospitalization, and incarceration. 
Among all AOT Participants to-date, 74% reduced or avoided PES contact through 
participation in AOT, 91% reduced or avoided time spent hospitalized, and 88% 
reduced or avoided time spent incarcerated. Among the group of 17 court ordered 
AOT participants, all 17 reduced or avoided PES contact, 14 reduced or avoided 
time spent hospitalized, and 15 reduced or avoided time spent incarcerated. 

Before participants enter AOT, many are in need of extensive support to stabilize. 
This is most easily achieved when participants willingly engage in services, but not 
all do. Nonetheless, AOT's practice of consistent and persistent outreach and 
engagement appears to have a positive effect on individuals, stabilizing them even 
when they are not engaged in treatment. Participants who were not able to 
function successfully in the community prior to AOT are often able to with the 
additional support of the AOT Team, and the majority of AOT Participants surveyed 
feel confident that they can reach their treatment goals, and feel hopeful about 
their future. 

The findings detailed in this report suggest overall reductions in negative outcomes 
for voluntary and court-ordered AOT Participants, as well as improvements in 
functioning and well-being. Given the stated goals of the AOT program to prevent 
decompensation and reduce cycling through acute services and incarceration, as 
well as the significant decrease in negative outcomes following an individual's exit 
from the program shown in this report, there is substantial evidence that 
indicates that implementation to date can be considered a success. In 
addition, analysis of the public costs associated with utilization of psychiatric 
emergency services, inpatient psychiatric hospitalization, and incarceration 
indicates significant estimated savings as a result of AOT implementation. 

Lessons Learned 

At the closing of this evaluation cycle, there are a number of promising tools and 
practices, employed by Care Team and the Citywide AOT Team alike, that have 
emerged since the implementation of AOT began in November 2015. Many of these 
can serve as models for best practices throughout the system of care, and they, 
along with other key insights and lessons learned, are summarized below. 

Engagement and Treatment Promotion 

• Persistence pays off. Staff, stakeholders, family members, and AOT 
participants all noted that consistent and persistent outreach by AOT staff 
leads to a higher likelihood of engagement, including with individuals who 
are initially unwilling to participate in treatment. This allows AOT to engage 
some of the hardest to serve individuals, many of whom the system of 
care has been unsuccessful in engaging in the past. This persistent 
outreach and high-touch service model is designed specifically to serve the 
small number of individuals who meet the legal eligibility criteria and are 
not otherwise consenting to necessary services. 
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• Meet people where they are. AOT staff spend a significant amount of 
time outreaching individuals on the street, in their residences, in hospitals, 
and jail. The AOT Staff's extensive outreach and engagement efforts allow 
for participants to be engaged at times and places when they are most 
readily able to engage, positively impacting participation in services and 
treatment. 

• High touch case management increases the likelihood of 
stabilization and successful engagement. The provision of intensive 
case management support for voluntary participants is critical for 
successful stabilization and long term engagement in treatment, providing 
consistent support to individuals when they need it most. 

• Let participants take the lead when possible. Engagement focused on 
participants' own treatment goals and future plans allows for respectful 
and client-centered support. However, in instances where individuals are 
not successful in engaging in voluntary care, AOT's provision of the court 
order option is vital for promoting stabilization. 

Family Support 

• Strengthening families can improve outcomes. Educating family 
members on mental health, what it means to have a mental illness, and 
how to strengthen participants' support systems can result in improved 
outcomes for program participants. 

• Allowing family members to refer their loved ones directly to AOT 
helps connect hard to engage individuals to services. Many family 
members reported feeling discouraged and even let down over their 
previous inability to help their loved ones access appropriate services 
throughout the system of care. By empowering family members to refer 
directly, individuals who may otherwise be disconnected from the system 
of care are more likely to be identified, outreached, and served. 

Relationship Building 

• Building a network of strong partnerships enhances the system of 
care. Strengthening relationships between agencies and organizations 
throughout the system of care allows for information and resource sharing, 
and enhances service provision throughout. 

• Collaboration throughout the system of care allows for the 
provision of individualized care and services and increases the 
likelihood of successful engagement. Through strong collaboration 
with providers across San Francisco, AOT staff have been able to efficiently 
connect participants to services tailored to their specific needs. When 
participants are supported by a collaborative network of providers in this 
way, they are less likely to disconnect from services, increasing the 
likelihood of stabilization, as well as successful and consistent engagement 
during recovery. 

Contact for More Information 

For more information on San Francisco's AOT program, or to make a referral, 

please visit www.sfdph.org/aot . To refer an individual by phone, please dial (415) 

255-3936 or, toll free, (844) 255-4097. 
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Appendix A 


San Francisco's Assisted Outpatient Treatment Fact Sheet 


Summary 

Assisted Outpatient Treatment, also referred to as "AOT or" "Laura's Law," was enacted in 2002 by California 
Assembly Bill 1421 and refers to court-ordered outpatient treatment for individuals who have a severe 
mental illness. In counties that choose to adopt the program, AOT allows for adults who meet certain 
requirements to request that the county mental health director petition the court to mandate treatment for 
individuals who have previously refused care and meet strict eligibility requirements. 

Overview of the Process 


Overview of Assisted Outpatient Treatment (AOT) Process • W&l 5345-5349. r ) 
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Who is Eligible for AOT? 


An individual must meet aN of the following criteria to qualify for AOT: 

1) Be at least 18 years of age; 

2) Suffer from a serious mental disorder (defined by W&I §5600.3 (b)(2) and (3)); 

3) Be unlikely to survive in the community without supervision, per clinical determination; 

4) Demonstrate a history of failing to comply with treatment (one of the following must be true): 

a) The person's mental illness has been a key factor in necessitating psychiatric hospitalization or 
mental health services while incarcerated at least twice within the last 36 months, not including 
the period immediately preceding the petition for AOT, or 

b) The person's mental illness has resulted in one or more incidents of serious and violent behavior 
toward himself or another in the last 48 months, not including the period immediately preceding 
the petition for AOT; 

5) Have been offered the opportunity to participate in treatment but failed to engage; 

6) Be substantially deteriorating; 

7) Be an appropriate match for AOT, meaning that AOT offers the least restrictive placement needed to 
ensure recovery and stability; 

8) Be unlikely to relapse or be subject to an involuntary psychiatric hold (5150) with AOT; and 

9) Likely benefit from AOT. 

Who Can Request a Petition for AOT? 

A request for AOT may be initiated by the following adults (age 18+): 

1) Any adult who lives with the individual with mental illness; 

2) A parent, spouse, sibling, or adult child of the individual with mental illness; 

3) The director of a mental health institution in which the individual with mental illness lives; 

4) The director of a hospital where the individual with mental illness is hospitalized; 

5) A licensed mental health provider supervising the treatment of the individual; or 

6) A peace, parole, or probation officer assigned to supervise the individual. 
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FAQs 


Who is AOT designed to help? 

AOT is designed to assist individuals who have a documented severe mental illness, who are not actively 
engaged in care, are in deteriorating condition, and have a history of failing to comply with treatment. AOT 
requires that individuals meet strict eligibility guidelines, as outlined above. 

How many people are expected to be eligible for AOT in San Francisco? 

SFDPH estimates participation to be fewer than 100 annually. (SFDPH currently provides mental health care 
for approximately 31,000 San Franciscans at 23 SFDPFI mental health clinics and programs, and 300 
contracted programs in the community. About 7,200 patients are treated each year at psychiatric 
emergency department at San Francisco General Hospital and Trauma Center. 

Does AOT help provide care for people with mental illness who are 
homeless? 

In some cases, homeless people will be eligible for AOT; in other cases they will not. AOT has strict 
eligibility criteria that apply regardless of whether an individual is housed. These criteria include the 
requirement that AOT be initiated by someone who knows the individual, either personally (family member 
or co-habitant) or professionally (mental health provider or peace, parole or probation officer assigned to 
supervise the individual), and that the individual not be actively engaged in mental health treatment. 

What are the individual's rights in the process? 

AOT strictly defines patient eligibility criteria in an effort to ensure appropriate application of the law and to 
protect individual rights. AOT provides at least two opportunities to engage patients in voluntary treatment 
prior to a court hearing. Additionally, AOT specifically defines the rights of the individual with mental illness 
who is subject to AOT, including adequate notice of hearings, to receive a copy of the court-ordered 
evaluation, to a court appointed public defender in the absence of private counsel, to be present at the 
hearing, to present evidence and call and/or cross-examine witnesses, and to appeal decisions. 

What is the difference between AOT and a 5150? 

A "5150" refers to Section 5150 of the California Welfare & Institutions Code and is an emergency hold in 
response to a psychiatric crisis, allowing for up to 72 hours of involuntary psychiatric evaluation and 
treatment of persons believed to be a danger to self, a danger to others, or gravely disabled by mental 
illness. AOT is a non-crisis process that allows for an adult that meets AOT criteria and declines voluntary 
treatment to be compelled by a civil court process to receive mental health care in the community. The goal 
of AOT is to support individuals with mental illness in the community in an effort to prevent future crisis. 

If someone does not comply with court-ordered AOT are they 
automatically subject to a 5150? 

No. Failure to comply with AOT alone may not be grounds for a 5150 involuntary hold or for a contempt of 
court finding. The criteria for a 5150 involuntary hold are already prescribed in state law and are no 
different for AOT Participants than for any other individual. In order to meet the criteria for a 5150 an 
individual must be at imminent risk for danger to self, at imminent risk for danger to others, or be gravely 
disabled (unable to care for basic needs such as food, shelter, and clothing) due to a mental illness. 

What are consequences of noncompliance with court-ordered AOT? 

If the treating mental health treatment provider believes that the individual is a danger to self, a danger to 
others, or gravely disabled and in need of involuntary treatment, the provider may initiate the 5150 process 
There are no additional enforcement mechanisms for individuals who do not meet 5150 criteria. Flowever, 
some jurisdictions that have implemented AOT have noted that court involvement itself can prompt some 
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patients to choose treatment, including medication. This has been called the "Black Robe Effect." 

Would AOT reduce the number of 5150s? 

The impact of AOT on 5150 involuntary holds is unknown. Once implemented, the data collection, reporting, 
and evaluation requirements under AOT would likely answer this question. 

How does AOT differ from SFDPH's existing Community Independence 
Placement Program? 

The Community Independence Placement Program (CIPP) is a voluntary program for individuals who have 
been subject to a 5150 involuntary hold and who meet the grave disability criteria required for 
conservatorship. Participation in the program is initiated in the hospital and participants are transitioned to 
community-based care. Participants agree to allow a conservator and the mental health court to work on 
their behalf to ensure that they adhere to their prescribed treatment plans, including medication adherence. 

AOT is court-ordered treatment initiated while the individual resides in the community. AOT provides a 
mechanism for family members and others who know the individual well to help engage an individual into 
treatment without requiring hospitalization or law enforcement. Individuals who meet strict eligibility 
requirements may be ordered by the court to receive mental health treatment. 

Can AOT mandate medication? 

No. State statute specifies that involuntary medication shall not be allowed absent a separate court order 
available only for individuals who are conserved due to their grave disability. 

How much will AOT cost? 

Other communities that have implemented AOT (Orange County, Nevada County) estimate the mental 
health treatment costs at $35,000-$40,000 per person per year. This does not include costs associated with 
the judicial system. Per State statute, no voluntary mental health programs may be reduced as a result of 
the implementation of AOT. 

How will the effectiveness of AOT be evaluated? 

Counties that implement AOT are required to collect and report key data to the State Department of Health 
Care Services for evaluation. At minimum the evaluation is required to include data that relates to number 
of individuals that receive services through AOT, engagement of AOT Participants in services, and key data 
points to measure the effectiveness of AOT as an intervention and engagement tool (e.g., hospitalization, 
contact with law enforcement, social functioning, and employment). 
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